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Abstract 

In a survey of educational researchers, school and district administrators, and policymakers, 1,818 
respondents rated educational practices and policies according to their influence on learning and 
assessability. Classroom practices, design and delivery of curriculum, and schoolwide practices were 
rated as more influential than federal, state, and district policies; however, the latter were rated as more 
assessable. In general, researchers and practitioners agreed about which practices and policies are 
influential but not about their assessability. Practices and policies rated both influential and assessable 
are the most feasible for use in educational planning and evaluation; those rated influential but less 
assessable call for development of new observational measures. This survey data can be used to guide 
local program development, assessment development, and in the monitoring of program implementation 
and evaluation of outcomes. 
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Effective Practices and Policies: 
Research and Practitioner Views 

Since A Nation at Risk (National Commission on Excellence in Education, 1983), public attention 
and efforts to reform schools have focused on improvmg students* achievement. Identifying practices 
and policies that enhance learning serves the interests of all stakeholders-parents, educators, and 
policymakers. Indeed, the education landscape of the 1980s and 1990s has been dominated by reform 
efforts. 

Research on what makes learning more effective provides a knowledge base to improve 
conventional practices. Research-based practices and policies can improve the capacity of schools for 
student achievement. Calls to upgrade the nation's teaching corps, to set standards, and to improve 
student learning underscore the need to bring what is known about learning into the reform agenda. 
Toward that end, this paper reports the findings from a survey of educators' and researchers' judgments 
about practices and policies that influence student learning and the degree of assessability of those 
learning influences. Several developments that provided the impetus for the survey are discussed in the 
next two sections. They show the need for research-based information on assessable policies and 
practices that work effectively. 
Raising Standards 

Standard setting has become the most visible activity in the educational reform movement. 
Beginning with the National Council of Teachers of Mathematics* Curriculum and Evaluation Standards 
for School Mathematics (1989), professional education organizations developed content standards that 
would influence curricula, assessment practices, and instructional delivery. States and school districts 
participated with the various standard-setting efforts in mathematics, science, English, language arts, 
history, and geography. Fuhrman and Elmore (1994) reported that in 1989 and 1990, the National 
Governors Association worked with former President Bush in "calling for, establishing, and promoting 
a set of national education goals" (p. 66). Federal government agencies embraced standard setting as a 



central approach to improving the educational enterprise. For example, the National Assessment of 
Educational Progress (NAEP) aligned its mathematics assessment to the NCTM standards. Both the 
Goals 2000: Educate America Act and the new Titie I legislation require tiiat eacli state establish 
standards for student achievement and center educational reform efforts around achieving tiiem (Gandal, 
1995). 

Publishers and testing companies have tailored their products to parallel tiie standards set by 
professional education groups and Goals 2000. Praxis UI, a component of a teacher assessment system 
developed by Educational Testing Service, is a high-stakes assessment used to determine whether 
provisionally licensed teachers should receive a continuing license. According to Danielson and Dwyer 
(1995), "A critical element [in tiie development of Praxis III] was the establishment of teaching standards. 
The resulting 19 criteria in 4 domains represent interrelated aspects of a complex performance" (p. 66). 

To reform tiie educational system more broadly, other standards are being identified. "Systemic 
reform" coordinates content, student performance, school delivery, and system delivery standards (Smith, 
Fuhrman, & O'Day, 1994). Content standards identify tiie knowledge and skills that students must 
master; student performance standards identify tiie degree of competency tiiat must be demonstrated for 
eadi content standard; and school delivery standards identify criteria indicatmg whetiier a school provides 
students witii tiie "opportunity to learn" tiie material identified in tiie content standards. Examples of 
school delivery standards might include: quality instructional materials needed to teach the content 
standards; alignment of tiie school curriculum witii tiie standards; and training staff to teach tiiem (Smitii 
et al., 1994). System delivery standards address tiie quality of tiie district, state, or federal systems' 
capability to educate all students as specified in tiie content standards. The identification and 
implementation of content, performance, and school and system delivery standards presumably result in 
systemic educational reform and improvement of student leammg. 
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School and system delivery standards are operationalized in terms of teacher behaviors, classroom 
practices, and schoolwide policies. It is therefore essential to measure the degree to which a school, local 
education association, or state education association has implemented a particular practice or policy. 
Setting school delivery standards, in particular, challenges researchers to identify practices and policies 
that are linked to improved student learning and that can be reliably and validly assessed. 
yp f p^ding the Nation's T eaching Corps 

During the 1980s state legislatures passed laws that imposed higher standards on teacher 
preparation and licensing. Evidence had accrued that prospective teachers typically enrolled in college 
classes that were less rigorous than those taken by students in noneducation degree programs. The 
Holmes Group Executive Board (1985) and the Carnegie task Force on Teaching as a Profession (1986) 
called for upgrading weak teacher education programs. They recommended tiiat teachers engage in a 5- 
year, rather tiian 4-year, course of study and set more rigorous course requirements. 

Paralleling these efforts, merit pay, career ladder programs, and systematic observation of 
mservice teachers were also employed. These approaches may have been of limited utility because they 
often were not linked to improved student learning. For tiiis reason, teacher evaluation systems and 
efforts at professional development more often include evidence of student learning as one of several 
criteria of teacher effectiveness (Sanders & Horn, 1993; Schalock, Schalock, Myton, & Girod, 1993). 

Surveys conducted by Impara and colleagues (Impara, Plake, & Pager, 1993; Impara, Plake, & 
Mecwm, 1994) have been used by professional educational organizations to identify teacher and 
administi:ator competencies, knowledge, and attitudes in assessment-related tasks. In one survey, 
approximately 1,700 school administirators (superintendents, elementary and secondary school principals) 
were asked to rate the frequency witii which tiiey performed 24 assessment tasks and the importance of 
these tasks. They also were asked to rate tiieir knowledge and need of 13 assessment-related practices. 
A second survey of approximately 300 administiiators and 555 teachers focused on knowledge of 
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classroom assessment. Teachers and school administrators received 35 items based on the Standards for 
Teacher Competence in Educational Assessment of Students (American Federation of Teachers. 1990). 
Survey respondents also were asked about their use of tests and their training in assessment. Both of 
these surveys were conducted to assist professional education organizations in formulating standards for 
educator competency in assessing students. 

Colleges of education, school districts, and state departments of education would benefit from 
empirical information linking pedagogy to improved student performance. Survey data as well as results 
of empirical studies could be used to develop courses of study for education majors, design professional 
development programs for preservice and inservice teachers, and aid in standard-setting efforts. 
Improving Student Educational Performance 

Raising academic standards alone may not result in increased student achievement. McDill, 
Natriello, and Pallas (1987) warn that high academic standards may seem unattainable to students at risk 
of school failure. Raising standards could result in some of these students academically disengaging and, 
eventually, dropping out of school. Meeting academic standards requires the implementation of a variety 
of instructional and schoolwide practices. 

Although recent developments in Congress suggest that support for standard-setting efforts is 
waning ("National History Standards," Feb. 1, 1995), efforts to raise low levels of student performance 
continue to receive backing from educators and the public alike. Given the limited resources of many 
school systems, identifying the most powerful influences on learning clearly is desirable. Models of 
educational performance and school learning aspire to identify most, if not all, of the important influences 
on students* learning and participation in school. Meta-analyses and research syntheses conducted over 
the past 15 years also provide a foundation that can guide the identification of effective educational 
practices and policies. 



Effective Praaices - 4 



ERIC 



8 



PREVIOUS RESEARCH SYNTHESES 

One basis for research-based reform is research synthesU. Walberg (1986) argues that research 

synthesis has . . brought a new level of scientific maturity to research on teaching" (p. 214). 

Quantitative syntheses have features that make their results more valid and accessible to consumers of 

research results. Walberg (1986) describes the advantageous features of research synthesis as follows: 

ntl explicitly applies scientific techniques and standards to the evaluation and fummary of 
research, it not only statistically summarizes effects across studies but also provides dmded 
descriptions of replicable searches of literature, selection of studies, metrics of study effect 
statistical procedures, and both overall and exceptional results with respect to expermiental 
conditions, context, or subjects, (p. 214) 

Increasingly, conclusions from quantitative syntheses that focus on education converge on a set 
of practices and policies that enhance educational outcomes. Lipsey and Wilson (1993) assert that the 
magnitude and direction of the effects reported in these syntheses, although small, are mostiy positive and 
have genuine practical significance. They further conclude that the results of such quantitative syntheses 
are . . more credible than those of conventional reviews" (p. 1). 
fi^vHi ple Infl yfint^^ on Learning 

Historically, research syntheses have focused either on a specific instructional practice or on 
several diverse practices in order to compare their relative effectiveness. The following syntheses fall 
into the latter category. 

Walberg. Schiller, and Haertel (1979) published one of the first quantitative syntheses of research 
on teaching. Hie autiiors collected reviews published between 1969 and 1979 on the impact or 
associationof instructional variables on students' cognition, affect, and behavior. Among the instructional 
practices synthesized were: time on task, mastery learning, psychological incentives, open versus 
traditional classroom practices, and use of advance organizers. Nearly two tiiirds of the effect sizes or 
correlations were positive, indicating tiiat many well-established educational practices enhance student 
achievement. 
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In a systematic examination of 19 reviews of teaching process-student outcome research, Waxman 
and Walberg (1982) found consistent and substantial agreement that cognitive engagement, motivationa! 
incentives, pupil involvement in learning, reinforcement, and management and classroom climate are 
positively associated with student learning outcomes. 

During the 1980s, Walberg and colleagues conducted syntheses of the influence of instruction, 
psychological environments, and student aptitudes on educational achievement. The syntheses focused 
on nine theoretical constructs hypothesized to be consistently related to educational learning: student age 
or developmental level; ability (including prior achievement); motivation; amount of instruction; quality 
of instruction; exposure to the mass media; and the psychological environments of the class, home, and 
peer group outside school. The results provided systematic evidence that the constructs are consistently 
correlated with learning (Walberg, 1984). 

Fraser and colleagues (1987) compiled an extensive review of research on factors related to 
school learning. They summarized results of over 2,000 bivariate studies that identified nine aptitudinal, 
instructional, and environmental factors that consistently exhibited strong influences on academic learning. 
Fraser et al. (1987) also synthesized 135 meta-analyses with school achievement as an outcome, as well 
as 92 meta-analyses of attitude outcomes; the studies spanned 50 years of research in the United States 
and elsewhere. Among the influences examined were contextual factors, including student and teacher 
characteristics, curriculum materials, facilities and equipment, home environment, and school climate. 

Wang ct al. (1990) rq?orted a content analysis of research literature on school learning. Seardi 
and selection procedures yielded 179 handbook and annual review chapters, comnussioned papers, and 
other authoritative reviews. Results confirmed the primacy of student characteristics, instruction, and 
home and community influences on academic learmng. More distal variables, such as state and district 
policy, proved less influential. More recenfly, Wang ct al. (1993) synthesized ratings of 61 research 
experts, 91 meu-analyses, and 179 handbook chapters and narrative reviews, compiling approximately 
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11,000 statistical relationships. Content analyses, expert ratings, and results from meta-afialyses revealed 
moderate to substantial agreement on the importance of the psychological, instructional, and home 
enviromneat variables. 

Results of these syntheses support the primacy of student characteristics, quality and quantity of 
instruction, and home and community influences on smdent learning. Hie dramatic pattern of overall 
positive results reported is characteristic of quantitative syntheses and meta-analytic reviews (Lipsey & 
Wilson, 1993). This pattern of results cannot be "explained as artifacts of meta-analytic techniques or 
generalized placebo effects" (Lipsey & Wilson, 1993, p. 1). 
.Specific Practices that Tmprove Learning 

Since the mid-1970s many quantitative research syntheses have been conducted on specific 
instructional practices. Among these are syntheses of computer aided/based instruction (Kulik & Kulik. 
1987; Ryan, 1991); programmed or individualized instruction (Bangert, Kulik, & Kulik, 1983); 
cooperative task structures (Johiison. Johnson, & Maruyaraa, 1983; Johnson. Maruyama, Johnson, 
Nelson, & Skon. 1981); smdent tutoring (Cohen, Kulik, & Kulik, 1982; Cook, Scruggs, Mastropieri, 
& Casto, 1986); behavioral objectives, reinforcement, cues, and feedback (Lysakowski & Walberg, 

1982) ; mastery learning (Guskey & Pigott, 1988); home environment (Graue, Wemstein, & Wa'berg, 

1983) ; technology-based instructional strategies (Shwalb, 1987; WUliams, 1990); reading instruction 
strategies (Pflaum, Walberg, Karegianes, & Ras^ier, 1980); whole-language approach (Stahl & MUler, 
1989); vocabulary instruction (Klesius & Searls, 1990; Stahl & Fairbanks, 1986); and bUingual instruction 
(WUlig, 1985). 

Summarizing across meta-analyses, a number of specific practices have consistentiy improved 
academic learning are degree of curriculum articulation and organization; abundant classroom materials 
that support the instructional program; maximized learning time; high student expectations; opportunities 
for students to give extended oral and v/ritten responses; degree of classroom engagement; student 
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participation in setting goals and making instructional decisions; opportunities for students to receive 
Intensive instruction in one-on-one or tutoring arrangements; engaging in cooperative learning; frequent 
assessment; and a home environment that supports learning (Fraser et al., 1987; Lipsey & WUson, 1993; 
Slavin. Kanveit, & Madden, 1989; Wang et al., 1993). 

The results of these syntheses provide evidence that many well-designed instructional practices 
have an effect on student learning. Although these syntheses did not consider the relative impact of 
several instructional practices simultaneously, they often did examine differences in the learners* contexts. 
It is beyond the scope of this paper to report the effects of the many educational treatments and praciices 
that have been studied using meta-analytic and synthetic methods. These results, however, served as the 
research basis for the extensive survey reported in this paper, as exp'.amed in a subsequent section. 

METHOD 

The purpose of the survey was to gather expert judgments from educational researchers and 
practitioners concerning the influence and assessability of a variety of educational practices and policies. 
The survey was designed to generate practical findings that could be used by educators in designing 
effective classroom, school, and disti-ict practices. In the winter of 1993, the maU survey was sent to 
approximately 3,100 school administrators, policymakers, and educational researchers. Survey recipients 
were asked to rate the degree of influence on learning and assessabUity of the 146 educational practices 
and policies. A five-step process was used to develop the survey: a theoretical framework of student 
learning was identified; items were selected and revised; two rating scales were constnicted; items war. 
classified into four categories; and three forms were compiled. 
Theoretical Framework 

Based on the results of prior quantitative syntheses (discussed above), a theoretical framework 
comprisijd of 228 Influences on student learning and organized within six theoretical constructs (State and 
District Governance and Organization; Home and Conununity Educational Contexts; School 
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Demographics. Culture. Climate. Policies, and Practices; Design and Delivery of Curriculum and 
Imtruction; Classroom Practices; and Student Characteristics) was adopted for purposes of this research 
(see Wang et al. [1993] for a detailed description of the framework and theoretical constructs). 

\^ fm Selectio n and Revision 

nxe 228 influences were used by Reynold^ Wang, and Walberg (1992) in a prior survey of 
regular classroom versus special education teachers, researchers, and school administrators. From the 
original pool of 228 items, a total of 146 items, those that focused on classroom and school practices and 
district, state, and federal policies, were selected for inclusion in the current surx'ey. Some items were 

rewritten tc improve tiieir clarity and to tailor them to the purposes of this study. 

rnnstnictio n Rating Scales 

Two three-point rating scales were constructed: degree of influence on learning, and assessability. 

Learning Influence was defined as providing students with opportunities to acquire important knowledge. 

skills, and attitudes. The rating scale was as follows: 

1 = Littie or no influence on learning 

2 = Moderate influence on learning 

3 = Strong influence on learning 

AssessabUity was defined as the extent to which the presence or absence of the policy or practice 
can be ascertained by objective observations, archived documents, or other means. TTie following rating 

scale was utilized: 

1 = Not assessable 

2 = Fairly assessable 

3 = Very assessable 

riassifvinF Items into F nnr Categories 

niree independent judges classified the 146 items into four categories: Classroom Practices and 
Policies (70 items); Schoolwide Practices and Polices (39 items); Curriculum Design and Delivery (16 
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items); and Federal, State, and District Policies (21 items). The items classified into each category are 

shown in Tables 2, 3, 4, and 5. 

Procedure 

Survey directions requested that respondents rate each item in terms of its influence on student 
learning and assessability. The 146 items on the survey were randomly divided into three forms, so that 
each recipient received no more than 50 items, drawn from all four of the item categories. Because the 
survey was to be administered to groups of researchers and practitioners, different background items were 
prepared for the two groups. Researchers were asked to identify their primary research interest (e.g., 
school administraUon, and curriculum studies) and their gender; practitioners were asked to identify their 
current position (e.g., principal, superintendent), the type of school administered (e.g., elementary, 
middie school), the location of schools or districts (e.g., all districts statewide, suburban), and their 
gender. Mail surveys were sent to all recipients in November 1993; nonrespondents were sent a follow- 
up survey in January 1994. 
Sample 

Eight samples were drawn from the following six organizations: American Association of School 
Administrators (AASA); American Educational Research Association (AERA) Divisions A 
(Administration), C (Learning and Instruction), and H (Evaluation); CouncU of C^iief State School 
Officers (CCSSO); CouncU of the Great City Schools (CGCS); National Association of Elementary 
School Principals (NAESP); and National Association of Secondary School Principals (NASSP). 
Membership lists were supplied by each organization. In two of the organizations (CCSSO and CGCS) 
every member was sampled because of the small universes. In the other fear organizations (AASA, 
NAESP, NASSP, and AERA) random samples without rq)lacement were drawn. 

Table 1 presents the number of recipients and percent return for each professional group and for 
the total sample. Returns are presented from the original mailing of the survey and tiie follow-up 
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maaing. For each of the three forms of the survey, chi squares were calculated to test differences among 
return rates by original and follow-up mailings for: males and females; membership in professional group 
for researchers (AERA Divisions A, C. and H); and membership in professional groups for praaitioners 
(AASA. NAESP, NASSP. CCSSO. and CGCS). Of the 12 chi squares calculated only one was 
significant, male versu? female researchers on Form 1 of the survey (x'=6.48, df=l. p<.01). These 
results suggest that the 42% of the sample that did not reply may not differ much from the 58% that did. 
allowing for change in address and other reasons. Because there was only an isolated significant 
difference, and in light of the moderately high return rate for the survey, the results from the original and 
follow-up mailing were combined. 



Insert Table 1 about here 



RESULTS 

Tables 2 through 5 show the overall mean ratings of influence and assessabUity. To identify 
relatively high-influence and high-assessabUity items, two somewhat arbitrary cutoff points were chosen. 
Items are designated with an "I" if their influence mean is above 2.33 (overall influence mean of all 
items) and an "A" if their assessabUity mean is above 2.19 (overall assessabUity mean). Practices and 
policies) with the greatest feasibUity are those that have high influence and high assessabUity. Hiose 
rated as having high influence but low assessabUity require attention ficom the research community 
because, although they are cuaentiy difficult to measure, they are linked to student learning. Practices 
judged to be of low influence are not discussed in detaU because they are of litfle value to reform efforts, 
findin g with jn ftftm Categories 

ria«mnm Practices . Table 2 shows the 70 Classroom Practice items ranked by the total sample 
according to degree of influence on learning. Items witii the highest influence ratings were those tiiat 
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focused on teacher as the agent of action; a rich, cognitively challenging classroom environment; and 
teacher-student interactions about the learning task. Of the items with the lowest influence ratmgs, few 
focused on the cognitive environment of the classroom; most centered on specific mediods of instruction, 
such as diagnostic-prescriptive techniques, academic tracking, cross-age tutoring, and instructional 
teaming. Other low-influence items focused on classroom climate variables, such as democracy 
Onvolving all students in classroom activities) and formality (expecting students to follow expliciUy stated 
rules). 
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The items rated most assessable by survey recipients represent tangible features of the classroom 
(particular instructional practices, resources, and grouping practices), such as size of instructional groups, 
well-organized lessons, frequent feedback, use of goal direction, computer-assisted instruction, frequent 
measurement of basic skills, and student collaboration. Explicit teachers* behaviors were rated as more 
assessable than items focusing on teacher style. Assessable items captured observable teacher actions and 
did not require judgments of teacher "encouragement," "enthusiasm," or other "style" variables. 

Seventeen of the 70 items (marked I and A in Table 2) were identified as having high influence 
on leamtag as well as high assessabUity. Tliese items are most feasible for immediate use in classrooms 
and schools; they focus on the central role of the teacher, observable teacher behaviors, and the academic 
focus C>-«-. providing corrective feedback, well-organized classrooms where students are appropriately 
challenged, and high expectations of content mastery). These items represent features of educational 
settings with which school tdministrators and practitioners are most familiar. 

Twenty-five of the 70 items were rated as having high infUxence but low assessability. These 
foCTised on less observable teacher actions, such as: promotion of metacognitive strategies; use of 
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cognitively challenging questions; use of rehearsal and elaboration of new concepts; and use of mstruction 
I to recognize and dispel student misconceptions. In contrast to the items rated as high influence and high 

assessabUity, these items require knowledge of cognitive principles of instruction as well as complex, 
inductive reasoning about teachers' actions. They are high-inference items. Judging whether ^ question 
I is cognitively challenging, for example, is more difficult than recognizing that a teacher is providing 

feedback. When the teacher's actions represent familiar educational pursuits, such as conducting a well- 
I organized lesson, giving feedback, or conducting assessments, the practices are judged as both influential 

and assessable. However, when they are cognitively based practices-such as correcting misconceptions, 
promoting metacognitive strategies, or asking questions that are cognitively challenging-the influence 
I ratings are high, but assessabUity ratings are low. These cognitively based practices are examples of the 

constructivist principles of learning and teaching, and the technology of assessing such teaching has not 
been well established. 

I rnrrirnhim Desig n and Delivery . Table 3 presents the 16 Curriculum Design and Delivery items 

ranked according to degree of influence on learning. Of the 16 items. 9 were identified as having a 
moderate or greater influence on learning. They focused on: aligmnent of curriculum content, 
I instruction, and assessment; tailoring tiie content to students' cognitive capabilities and prior knowledge; 

and avaUabUity of materials, instructional activities, assessments, and equipment. TTie seven items rated 
as having less influence on learning were characterized by: the role of student interests and cultural 
I diversity; avaUability of classroom aides and efficient use of classroom space; teacher encouragement of 

self-regulated learning strategies; use of written records to monitor student progress; and curriculum units 
I structured around key discipline4)ased concepts. Curriculum-related practices were perceived as 

I influential when they are sensitive to students' cognitive needs, but less influential when they adapt to 

student interests, preferences, and cultural backgrounds. The physical enviromnent of the classroom was 
I perceived as less influential on learning. Further, matching curriculum content to a chUd's cognitive 
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capabilities was perceived as more influential than promoting self-regulated learning or metacognitive 
strategies. 
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The 11 items judged as relatively assessable focused on tangible features of the instructional 
environment, such as: use of objectives; inclusion of assessments; explicit classroom rules and 
procedures; alignment among goals, instruction, and evaluation; the difficulty level of the materials; 
availability of materials and activities for instructional groups of different sizes; and avaUabUity of 
classroom aides. Those items judged as less assessable focused on students' past experiences, interests, 
and cultural backgrounds; well-configured classroom space; and teacher development of self-regulated 
learning strategies. 

Eight of the 16 items were rated both influential and assessable. These items generally reflected 
the availabUity of materials, assessments, and equipment as well as the features of the materials, 
especially their cognitive difficulty. 

Only one item was rated as influential and not assessable: use of materials tiiat reflect student 
experiences. Making a judgment about, for instance, whether an art unit on the Renaissance matches the 
academic and social experiences of a classroom of students is complex and subjective. Hie complexity 
of the judgment is heightened by having to assess the suitability for a classroom of students ratiier than 
for a smgle student. 

Schoolwide Praf^»r.es and Policies . Of the 39 items in this category (see Table 4), 21 were 
identified as having a moderate or greater influence on learning. The focus of tiiese more influential 
items was: a safe, orderly, and positive school climate; site-based management practices; parent 
involvement programs; guarding of student instructional tune; shared curriculum decision-making among 
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staff and administrators; features of effective schools programs, especially an academic school climate; 
low staff turnover and alienation; and small school size. Hie practices and policies identified as 
influential coincide with many of the beliefe fostered by the effective schools movement about what 
improves student performance. 



Insert Table 4 about here 

•nre 18 items rated as having less influence were: policies to influence students' outK)f-school 
behaviors; schoolwide activities to influence student self-esteem, attitudes, and social condua O-e.. 
discouraging delinquent and criminal behavior, encouraging friendships over clique formation, increasing 
student occupational aspirations, promoting motivation toward lifelong learning). In addition, policies 
on schoolwide attendance, grading, academic progress, and suspension and expulsion were rated as less 
influential. 

Sixteen of the 39 items were rated as moderately or highly assessable. Assessable items were 
those that were easy to judge as presem or absent-a safe, orderly school climate; school district 
decentralization; small school size; and low staff absenteeism. The remaining 23 items, judged less 
assessable, were those that would require evidence of consensus, positive attitudes, student use of out-of- 
school time, and other practices and policies that are less observable (e.g., schoolwide activities to 
promote positive, nondisruptive behaviors; a positive attitude toward school, teacher, and subject matter; 
and self-esteem and self-confidence). 

Twelve of the 39 items were rated both influential and assessable. Hiese items involve explicit 
policies and practices that could readUy be discerned and in some cases quantified (e.g., small school 
size; low staff absenteeism; presence of schoolwide policies for parent involvement, recognition of 



Effective Practices - 15 



academic achievement, or discipline). Most of these items represent popular approaches to improving 
school effectiveness. 

Nine of the 39 items were rated as highly influential but low in assessabUity. Most of these items 
focused on policies that influence school climate. TTiey require inferences about atUtude toward school, 
teachers, and subject matter; perseverance on learning tasks; and aspirations. 

pRderal. State, anH District Policies . Of the 21 items m this category (see Table 5), only 4 were 
rated as having a moderate or greater influence on learning. These items were: central office support; 
board of education support; academic course and unit requirements; and higher per-pupU expenditures. 
The remaining 17 items, rated as less influential, were characterized by: levels of categorical funding; 
assessment requirements; provision of student social services; length of school day and year; teacher 
licensure requirements; contractual limits on class size, classroom aides' activities, and teachers' after- 
school meetings; efficient transportation system; school district size; and school district decentralization. 
The more influential policies are those that impinge most directly on students' lives-academic 
requirements and funds for instruction. 



Insert Table 5 about here 



Only 2 of the 21 items were rated difficult to assess-limited school district bureaucratization, and 
central office assistance and support. Central office assistance and support would have been classified 
as assessable if the mean rating had been .01 higher. Nearly all of the policy items were judged at least 
moderatdy assessable because the presence or absence of a stated federal, state, or district policy can be 
readily detected. 
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Tte three item that were both highly influential and highly assessable were: board of education 
support; academic course and unit requirements; and higher per-pupil expenditure. All three of these 
items represent policies set at the district and state level. 

Central office assistance and support for school programs, as well as limited school district 
bureaucratization, were the only items judged as influential but not assessable. Limited school distria 
bureaucratization is difficult to quantify, and hence to measure. 
Mftan Influence and Assessahilitv bv Item Category 

Table 6 shows the mean influence and assessability ratings for each item category as well as a 
grand mean for influence on learning and assessabUity. Based on the grand mean for influence, those 
categories judged more influential were Classroom Practices and Schoolwide Practices and Policies. 
Considered less influential were Curriculum Design and Delivery and Federal. State, and District 
Policies. 



Insert Table 6 about here 



Those categories that deal with matters proximal to students are judged more influential; those 
that deal with matters distal to students are judged less so. Interestingly, altiiough Federal, State, and 
District Policies were considered least influential, they were considered the most assessable category. 
Curriculum Design and Delivery. Classroom Practices, and Schoolwide Practices and Policies were 
considered to be less assessable. Indeed, the presence or absence of particular federal, state, or district 
policies is readily quantified, whereas assessment of classroom practices, schoolwide practices and 
policies, and design and delivery of curriculum involve making higher-level inferences and are more 
challenging to assess. 
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Relation of Influence and A ssessabilitv for Groups 

The product-moment correlation between influence and assessability for the entire sample was 
.20. This provides little evidence of congruence between influence on learning and assessabUity. 

Item means were calculated for the combined researcher group (AERA Divisions A, C, and H) 
and the practitioner group (AASA, CCSSO, CGCS, NAESP, and NASSP). Table 7 shows very high- 
agreement between the researchers and practitioners in the ratings of influence and assessability. 
Researchers, however, found no congruence between influence on learning and assessabUity, whereas 
practitioners found moderate congruence. 
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Differences among Groups 

Table 8 presents the results of 10 one-way analyses of variances (ANOVA). The independent 
variable was the eight professional groups (AERA-Division A; AERA-Division C; AERA-Division H; 
AASA; CCSSO; CGCS; NAESP; and NASSP). The dependent variables were the influence and 
assessabUity mean ratmgs for each of the item categories, a total mean influence rating, and a total 
assessabUity ratmg. The five analyses of variance for influence (the four item categories and the total) 
were significant (p < .0001). For each of the five analyses of variance for influence, the three researcher 
groups (AERA Divisions A, C, and H) rated the items as less influential than did the five practitioner 
groups (AASA, CCSSO, CGCS, NAESP and NASSP). 



Insert Table 8 about here 



Effective Practices - 1 8 2 g 

ERIC 



Three of the five ANOVAs for assessability were significant (p< .0001): Classroom Practices; 
SchoolwidePraaices and Policies; and Federal. State, and District Policies. The researcher groups rated 
the Federal. State, and District Policies items as more assessable than did ♦he practiUoner groups. In 
general, elementary and secondary school principals rated the Classroom Practices item category as well 
as the Schoolwide Practices and Policies item category as more assessable than did the other groups of 
practitioners and researchers. Total assessability also was significant (p < .005); again, elementary and 
secondary school principals rated the items as more assessable than did the other groups. 

The large sample sizes in each professional group made small differences between the eight 
professional groups mean ratings on item categories statistically significant. TTius. the significant 
differences reported among the groups must be interpreted somewhat cautiously. 

The differences among professional groups are illustrated in Figures 1 and 2. As seen in Figure 
1, researchers rated the four item categories as less influential than the practitioner groups. Both 
researchers and practitioners rated tlie Federal, State, and District Policies as the least influential item 
category. 



Insert Figure 1 about here 



In Figure 2. researchers and most practitioner groups rated the assessabUity of Federal. State, and 
District Policies more highly than the other categories. Curriculum Design and Delivery items were 
generally rated as more assessable than the Classroom Practices and Schoolwide Practices and Policies 
item categories. Professional groups whose memberships are composed of school principals (NAESP and 
NASSP) tended to rate classroom, curriculum, and schoolwide practices as more assessiible than did 
groups composed of district administrators (AASA) and state school officers (CCSSO). 
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Insert Figure 2 about here 



APPLYING THE SURVEY DATABASE 
From the survey data, a framework emerged that can guide reform efforts. Based on each survey 
item's influence and assessability ratings, we identified items that Ulustrate how survey results can be used 
to: guide the development of sitfr-specific programs; monitor program implementation; evaluate program 
outcomes; and guide teacher education and professional development. 
Site-Specific Program and Assessm ent Development 

Site-specific program development can be based on the classroom practices, features of 
curriculum design and delivery, and schoolwide practices enumerated in the survey items. By reviewing 
the survey items that were rated as exerting a high degree of influence on student learning, local 
practitioners can identify the practices and policies that meet their particular needs and are likely to 
increase student leammg. For example, a school district that wants to introduce student collaborative 
grouping as an alternative to whole-class instruction would consult the Classroom Praaices item category 
and fmd three practices relevant to their mterest: use of small instructional groups; cooperative learning 
groups; and peer tutoring. These three practices were also rated as highly assessable. Based on research 
evidence and expert judgments, these classroom practices can be recommended as worthwhUe investments^ 
likely to enhance student learning. 

Several Curriculum Design and Delivery items can inform the use of collaborative student groups. 
For example, one item focuses on the importance of having a variety of curriculum materials avaUable 
for use with instructional groups of different sizes. Two additional items focus on tailoring classroom 
materials to students' developmental and ability levels. TTiese items were rated as influenUal and 
assessable, as were the Classroom Practice items. Thus, botli classroom practice and the design of 
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curriculum materials can be informed by the application of survey results to a specific site. Further 
program development could be based on items from the Schoolwide Practices and Policies category and 
the Federal, State, and District Policies categories. 

Hie example above demonstrates how a local education agency might use the survey database to 
select effective classroom practices (collaborative student groups) and design the curriculum materials 
taUored to the specific children and context being served. In addition to contributing to program 
development, the survey results can guide assessment development at local sites. Many survey items 
were rated as high influence but low assessability (see Taoles 2 through 5). Local practitioners may wish 
to develop assessments for items tiiat are high influence and low assessabUity. For example, several 
items in the Classroom Practice category focus on instruction based on cognitive principles (e.g., teacher 
use of scaffolding, teacher use of rehearsal and elaboration of new concepts, teacher use of instruction 
to recognize and dispel student misconceptions). All tiiese items were rated as high on influence and low 
on assessabUity. If a district or school chooses to develop a constructivist program based on cognitive 
principles of learning and instruction, it may be necessary for tiie local site to develop new assessments. 
The survey results can identify which practices and policies have readily available measurements and 
which would require die construction of new measures. 
Monitoring Program Imp lementation and Evaluation of Outcomes 

Some survey items can be used to monitor program implementation and outcome4)ased 
evaluations. Below are selected items that could be C'sed to monitor a program implementation. These 
items represent general implementation processes that apply to most programs, 
(^li^f^grftom Practices. 

Well-organized and well-planned dass activities 
Presence of a variety of instructional activities and content 
aearly presented academic, social, and attitudinal program goals 
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Alignment among goals, content, instruction, assignments, and evaluation 

Availability of materials and activities for use with whole classrooms, small groups, or 
one-on-one instruction 

Well-equipped classroom 

Teacher use of efficient and wsll-communicated routines, rules, and procedures 
<;fh9r>lwt< ;1ft Practices and Processes . 

Safe and orderly school climate 
Low staff absenteeism 
Federal. State, and District Policies . 

Board of education support for school programs 
The items specified above can be used to monitor the implementation of a variety of programs. 
These generally would have to be supplemented by survey items focused on particular features of the 
program being implemented. For example, the implementation of a program to establish an academically 
challenging classroom and school environment might be monitored using survey items, such as: 
Teachers provide frequent, correaive feedback 

Use of materials tailored to students' different abilities and developmental levels 
Use of materials that include assessments and diagnostic tests 
Schoolwide promotion of increased student time on task 
Schoolwide emphasis on and recognition of academic achievement 

Survey items can also be rewritten as evaluation outcomes and used, along with other criteria, 
to judge program success or failure. For example, the Classroom Practices item "use of goal direction 
(specific and explicit objectives of learning activities" can be rewritten as the following evaluation 
outcome: "Teachers use lesson objectives to promote students* sense of direction and purpose in their 
mstructional activities." This outcome can be operationalized through tiie use of indicators such as "95% 
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of classroom lessons observed will include an explicit presentation of learning objectives"; "a review of 
teachers- written instructional plans reveal objectives for each lesson"; and "65% of students in a given 
classroom will report understanding the purpose of their instructional activity." Thus, survey items are 
a catalog of practices and policies that can be rewritten as outcomes and indicators for use in summative 
evaluations. 

Tftacher Ed \ratifTn i'"^ FrA^S'^"^^ Development 

Survey results also can be us::1 by teacher educators to plan a course of study for preservice 
teachers. TTiis survey identifies pedagogical practices that are linked to student learning. Based on the 
survey results teacher educators can determine which of the practices, rated as influential, should be part 
of a course of study for education majors. Likewise, professional development courses for inservice 
teachers can be designed around constellations of influential practices and policies. In keeping with recent 
changes in designing professional development experiences, staff developmem should not be fragmented 
or piecemeal. Staff development efforts should be based on a strategic plan for change (Fullan. 1991; 
Sarason. 1990; Sparks, 1995). The survey results combine research-based information and expert 
judgments on 146 practices and policies at the classroom, curricular, school and district, state, and federal 
levels. These results provide a coherent knowledge base that can guide a school district or a school's 
professional development efforts. As results-driven education and systemic reform continue to exert 
influence on educational change, the database reported in this paper can provide credible guidance for 

professional development. 

Below is a list of topics tiiat could be used by teacher educators and staff developers to enhance 
preservice and inservice teachers' knowledge and skills. The topics listed below are organized by item 
categories. Each topic was identified using one or more survey items that were rated as botii influential 
and assessable. 
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Claj^sroom Practices Item Category . 

Setting high academic expectations for all students 

Developing a well-organized classroom 

Use of student collaborative groups to promote learning 

Providing effeaive written and oral feedback 

Curriculum Design and Delivery Item C ategory. 

Adapting curricula, instructional activities, and materials to students' cognUive levels and 
prior knowledge 

Aligning goals, curriculum content, instructional events, and assessments 

Creating classroGms with a variety of materials and equipment that meet the needs of 
diverse student populations 

Development and communication of classroom routines, rules, and procedures 
< j:chnolwide Practical and Poli cies Item Category. 
Guarding student instruaional time 
Establishing an effective parem involvement program 

Creating a school climate that is safe, orderly, positive, and academically rigorous 
Shared decision making among school staff, administrators, and parents 
Federal. State, and District Polic ies Item Category. 

Informing teachers about district and state policies 
Defining the role of boards of education in re^dtalizing school programs 
Allocating junds for cost-effeaive instruction 

CONCLUSIONS 

Based on Uie results of this survey, there is a shared view held by the educational research and 
prartitioner communities about which practices and policies are most likely to influence student leammg. 
The finding of a correlation of .87 between researchers' and practitioners' judgments of influential 
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practices and policies suggests that educational researcli findings are widely disseminated and have 
Influenced educational practitioners* beliefs about effective praxis. 

Although there is high agreement about which practices and policies influence student learning, 
there is less consensus about their assessability . The order of the item categories when ranked from most 
to least influential is: classroom practices, schoolwide practices and policies, curriculum design and 
delivery, and finally, federal, state, and district policies. Ranked from most to least assessable, the item 
categories are: federal, state, and district policies, design and delivery of curriculum, classroom practices, 
and schoolwide practices and policies. Federal, state, and district policies were the least influential of 
the item categories, but the most assessable. 

The assessability of the practices and policies is a criterion that must be considered as schools, 
districts, and states develop and implement strategies to improve student and teacher performance. 
Assessability, however, cannot be the sole criterion when strategies to improve academic performance 
are being designed. In those cases where an influential practice or policy is judged to have low 
assessabUity, resources need to be allocated to develop a valid and reliable assessment. Low assessabUity 
of a practice or policy only reduces the rapidity with which the practice or policy can be implemented, 
it should not eliminate the practice or policy from use. Local assessment development combined with 
local program development ensures that site-specific reform efforts can be taUored to the needs of 
particular communities. 

During the past 15 years, three waves of educational reform have altered educational practice and 
policy at the national, state, and local levels (Murphy, 1990). We have moved beyond the first and 
second waves of reform, which focused on changing the centralized educational system and empowering 
parents and teachers through shared decision making and collaboration. Hie third, an ongoing wave of 
reform, is focused on increasing student learning. The success of this effort can be facUitated through 
the application of effective practices and policies that enh-uice student learning and teacher performance. 
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nie survey results presented in this paper provide new knowledge that is direcUy relevant to the current 
wave of reform. 

The combination of research-based survey items and expert judgments reported in this survey 
provides one basis for systemic educational reform. Each survey item was based on a correlation or 
effect size between a particular practice or policy and student achievement. Thus, each item has criterion- 
related validity. The influence and assessability ratings for each item provide expert judgments from the 
practitioner and research communities. The survey results can guide site-specific efforts to reform 
classroom, school, district, and state practice and policy. Item influence and assessability ratings can 
inform local program development and identify the need for assessment development. Items can be 
selected to monitor program implementation. Items can also be revised to serve as outcomes in 
summative evaluations. In each case, the features of local education context determine which survey 
items are most useful to their reform efforts. Local practitioners must judge which of the practices and 
policies are viable given the economic, philosophical, political, and cultural climate of tiieir communities. 

Changing the educational system at the state, district, and school levels is complex. Teachers and 
school administrators must implement changes at the level of curriculum, classroom instruction, school 
organization, and governance in a manner that aligns school, district, and state goals (Schwartz, 1993). 
Strategies must be developed that guarantee all students will have an opportunity to master an 
academically rigorous curriculum. Effective instructional metiiods and sensitive, fair assessments must 
be identified or developed. Professional development must be designed and delivered to ensure that 
preservice and inservice teachers have the content and pedagogical knowledge and skills required for such 
changes. All these components of change-alignment of goals, new curricula, instructional strategies, 
assessments, methods of school governance, and professional development-must then be implemented 
and evaluated to judge tiie success of the reform efforts. Multidimensional reform efforts must be 
taUored by local education agencies to meet the needs and characteristics of thek students, faculties, and 
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communities. In order for schools, districts, and states to construct such reform efforts, they must have 
I access to new knowledge about effective practice and policy. Practitioners cannot upscale their local 

programs unless there is information avaUable about the established success of educational practices and 
policies. Results of this survey are a rich source of such information. 
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Table 1 



Original and Follow-Up Survey Returns and Total Survey Recipients 



Proiessionai ijrroup 


Survey Recipients 
N 


Original 


Returns (%) 
FoUow-Up 


Total 


AFP A-Division A 


498 


235 (78.1) 


66(21.9) 


301 (60.4) 


APRA-Division C 


500 


237 (79.0) 


63 (21.0) 


300(60.0) 


AERA-Dtvision H 


499 


245 (80.3) 


60 (19.7) 


305 (61.1) 


AASA 


546 


227 (73.9) 


80 (26.1) 


307 (56.2) 


ccsso 


57* 


22 (56.4) 


17 (43.6) 


39 (68.4) 


CGCS 


43 


21 (77.8) 


6 (22.2) 


27 (62.8) 


NAESP 


500 


201 (67.9) 


95 (32.1) 


296 (59.2) 


NASSP 


498 


178 (73.3) 


65 (26.7) 


243 (48.8) 


Total 


3141 


1366 (75.1) 


452(24.9) 


1818 (57.9) 



Virgin Islands, and Manila/Philippines. 
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Table 6 



Overall Mean Influence and Assessability by Scale 




Mean Influence 


Mean Assessability 


Classroom Practices 


2.39 


2.16 


Design and Delivery of Curricu- 


233 


2.25 


lum 








2.36 


2.15 


Schoolwide Practices and Policies 








2.10 


2.35 


Federal, State, and District Policy 








234 


2.19 


Grand Mean 
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Researcher 
Influence 

Researcher 
Asscssability 

Practitioner 
Influence 

Practitioner 
Asscssability 



Table? 

Pearson Product Moment Correlations for Researcher 
and Practitioner Innucacc and Assessability Ratings 



Researcher 
Influence 



100 



-.03 



.87 



.47 



Researcher 
Assessability 



Practitioner 
Influence 



Practitioner 
Assessability 



1.00 



-.10 



.68 



1.00 



.52 



1.00 
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Tables 

One-Way Analyses of Variance Comparing Ratings of Professional Groups 



Mean Square 

Dependent Variables N Sample Residual F(p) 



Influences 

Classroom Practices 1823 

Curriculum Design and Delivery 1823 

School wide Practices and Policies 1823 

Federal, State, «fe District Policies 1 823 

Total 1823 

Assessabilitv 

Classroom Practices 1823 

Curriculum Design and Delivery 1823 

School wide Practices and Policies 1823 

Federal, State, & District Policies 1 823 

Total 1823 



3.21 .10 32.65 (.0001) 

2.34 .14 16.27 (.0001) 

5.16 .11 47.30 (.0001) 

3.72 .14 27.23 (.0001) 

3.65 .08 46.08 (.0001) 

.75 .14 5.55 (.0001) 

.31 .17 1.77 (.089) 

.75 .13 5.77 (.0001) 

3.04 .17 18.36 (.0001) 

.29 .09 3.15 (.003) 
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